fix(manual-qa): async-dispatch metrics fixes + full tokenomics parity in bundle - #49
Merged
Merged
Conversation
…s into bundle Claude Code's current build dispatches the Agent tool asynchronously (an immediate "launched" acknowledgment, with real per-dispatch usage delivered later via <task-notification> messages on the orchestrator's own transcript) instead of the synchronous, blocking dispatch this pipeline was designed around. Left as-is, that silently breaks per-TC metrics for any project on this bundle re-run today: benchmark-tc-hook.mjs's real-time PostToolUse capture never sees usage data, so tcs[] and tokens_by_agent go empty and the whole session's tokens get misattributed to test-run-lead. The same async-dispatch behavior also makes the SubagentStop hook fire on every orchestrator pause while a background task is still running, not just on the real test-reporter completion, repeatedly wiping in-flight trace data before a suite finishes. A related but separate issue: the ccusage pre-snapshot taken at SessionStart is usually captured before Claude Code's own usage logs have indexed the brand-new session at all, so session-level token/cost figures silently fall back to summing every session on the machine instead of just this run's. All three were root-caused and fixed against real runs in qa-project this week, then synced into elitea-testing/qa-challenges (2026-07-31) — this change ports the same three fixes into the bundle so any project installing manual-qa from here on gets them too: - scripts/resolve-subagent-traces.mjs (new): backfills the tc-trace from the orchestrator transcript's own <task-notification>/<usage> blocks instead of trusting benchmark-tc-hook.mjs's real-time capture. Wired into benchmark-stop, which now also resolves the true parent transcript for SubagentStop firings (this bundle copy didn't have that resolution at all yet, unlike elitea-testing/qa-challenges — ported as a prerequisite). - benchmark-stop: re-derives real subagent identity from the payload's own agent_type field (empty string on a premature firing, "test-reporter" only on the genuine completion) instead of trusting the SubagentStop matcher, which doesn't reliably filter on affected Claude Code builds; elapsed-time-since-last-dispatch kept as a secondary net for payload shapes without that field. Debug logging added (gitignored via the existing *.log rule) so any future payload-shape drift is directly observable instead of re-diagnosed from scratch. - scripts/ccusage-wait.sh (new) + its use in benchmark-session-start and benchmark-preflight: retries ccusage session --json for up to 60s of real elapsed time (not a fixed attempt count — an earlier count-based attempt was disproven by real timing data) until this session's own entry appears, instead of a single immediate un-retried call. benchmark-preflight now also verifies session-start's snapshot is actually scoped before reusing it, rather than propagating an unscoped give-up forward. Deliberately NOT touched: build-run-metrics.mjs's turns/tokens_by_model/ subagent_dispatches instrumentation is still absent from this bundle copy (a separate, previously-deferred porting decision, see project memory) — transcript_path is resolved in benchmark-stop only as far as resolve-subagent-traces.mjs needs it, not plumbed into build-run-metrics.mjs itself. hooks/README.md's file count updated (10 -> 12) for the two new scripts; no other docs/behavior changed. Verified end-to-end in isolated scratch directories against real captured data: replayed real SubagentStop payloads (captured live in qa-project) to confirm the premature-firing guard skips correctly and proceeds on the genuine completion; ran resolve-subagent-traces.mjs against a real orchestrator transcript and confirmed its output matches that run's actual Performance Metrics table exactly; ran ccusage-wait.sh against a real, already-indexed session id and confirmed immediate scoped match; ran the full benchmark-stop -> resolve-subagent-traces.mjs -> build-run-metrics.mjs chain and confirmed real per-TC tokens/tool_uses in the output with turns/tokens_by_model correctly still absent (out of scope for this change).
…ics parity
The bundle carried a leaner, older build-run-metrics.mjs (615 lines, no
turns/subagent_dispatches/orchestrator_cost_pct/tokens_by_model/
cache_read_share_pct/scopedModelsUsed) while its own
knowledge/metrics-format.md already documented and gave a worked example of
the FULLER schema, and benchmark-stop's own comment explicitly flagged the
gap ("this bundle copy doesn't plumb transcript_path through to
build-run-metrics.mjs at all yet ... not applicable here until that feature
gap is ported"). Docs and code had drifted apart; this closes that gap so
a fresh `npx github:arozumenko/sdlc-skills init --bundle manual-qa` install
actually produces what its own documentation already promises.
Diffed the bundle's build-run-metrics.mjs against elitea-testing's current
copy (freshest reference, post yesterday's async-dispatch sync) plus
build-tokenomics-report.mjs/build-tokenomics-html.mjs/compare-runs.mjs for
completeness: those three were already fully at parity (compare-runs.mjs
doesn't even exist in elitea-testing/qa-challenges — the bundle is ahead
there), narrowing the real gap to exactly one file.
- build-run-metrics.mjs: ported the full feature set from qa-project's
current copy (already verified feature-identical to elitea-testing's,
and already using this bundle's own env-var-based PROJECT_DIR convention,
so the port needed no structural adaptation) — countTurns() from a
transcript_path argument, subagent_dispatches, orchestrator_cost_pct
(token-share proxy, documented as such), tokens_by_model +
cache_read_share_pct (keyed off ccusage's own scoped modelsUsed[], not
hardcoded model ids), and models_used.
- benchmark-stop: now actually passes transcript_path through to
build-run-metrics.mjs (resolved in the previous commit for
resolve-subagent-traces.mjs's sake, but deliberately withheld from this
call since the old build-run-metrics.mjs couldn't consume it yet).
Not touched: knowledge/metrics-format.md, hooks/README.md, and
hooks/templates/*.template.md were all already accurate/complete for this
schema (checked field-by-field) — no doc changes needed, only the code
producing what they already promised.
Verified end-to-end in an isolated scratch directory: replayed real
SubagentStop payloads (captured live in qa-project) through the full
benchmark-stop -> resolve-subagent-traces.mjs -> build-run-metrics.mjs
chain and confirmed real turns/subagent_dispatches/tokens_by_model/
cache_read_share_pct/models_used/tokens_by_agent all populate correctly
alongside the exact real per-TC tokens/tool_uses from a real captured
transcript.
arozumenko
approved these changes
Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Claude Code's current build dispatches the
Agenttool asynchronously (an immediate "launched" acknowledgment, with real per-dispatch usage delivered later via<task-notification>messages on the orchestrator's own transcript) instead of the synchronous, blocking dispatch the manual-qa metrics pipeline was designed around. Left as-is, this silently breaks per-TC metrics for any project on this bundle:tcs[]/tokens_by_agentgo empty, theSubagentStophook fires prematurely on every orchestrator pause (wiping in-flight trace data), and the ccusage pre-snapshot race can silently fall back to summing every session on the machine instead of just the current run.All of this was root-caused and fixed against real runs this week in
qa-project, verified, then synced intoelitea-testing/qa-challenges(2026-07-31). This PR ports the same fixes into the bundle so any project installingmanual-qavianpx github:arozumenko/sdlc-skills init --bundle manual-qagets them too — plus closes a separate, longstanding gap where the bundle'sbuild-run-metrics.mjshad fallen behind its own documentation (knowledge/metrics-format.mdalready describedturns/tokens_by_model/etc. as part of the schema; the code didn't produce them).Commit 1 —
fix(manual-qa): port async-dispatch trace recovery + hook-timing fixes into bundlescripts/resolve-subagent-traces.mjs(new): backfills the tc-trace from the orchestrator transcript's own<task-notification>/<usage>blocks instead of trustingbenchmark-tc-hook.mjs's real-time capture (which async dispatch makes unreliable). Wired intobenchmark-stop, which now also resolves the true parent transcript forSubagentStopfirings (this bundle copy didn't have that resolution at all yet, unlikeelitea-testing/qa-challenges).benchmark-stop: re-derives real subagent identity from the payload's ownagent_typefield (empty on a premature firing,"test-reporter"only on genuine completion) instead of trusting theSubagentStopmatcher alone; elapsed-time-since-last-dispatch kept as a secondary net. Debug logging added (gitignored).scripts/ccusage-wait.sh(new) + use inbenchmark-session-start/benchmark-preflight: retriesccusage session --jsonfor up to 60s of real elapsed time (not a fixed attempt count — disproven by real timing data) until the session's own entry appears, instead of one un-retried call.Commit 2 —
feat(manual-qa): bring bundle's build-run-metrics.mjs to full tokenomics paritybuild-run-metrics.mjs: ported the fuller feature set already present inelitea-testing/qa-challenges/qa-project—countTurns()(via thetranscript_pathresolved in commit 1),subagent_dispatches,orchestrator_cost_pct,tokens_by_model+cache_read_share_pct(keyed off ccusage's own scopedmodelsUsed[]),models_used.benchmark-stop: now actually passestranscript_paththrough tobuild-run-metrics.mjs(resolved in commit 1 forresolve-subagent-traces.mjs's sake only, until this commit could consume it too).build-tokenomics-report.mjs/build-tokenomics-html.mjs/compare-runs.mjsagainstelitea-testingfor completeness — already at parity (compare-runs.mjsdoesn't even exist there — bundle is ahead). Docs (knowledge/metrics-format.md,hooks/README.md, templates) were already accurate for this schema — no doc changes needed, only the code catching up to what they promised.Test plan
SubagentStoppayloads (captured live inqa-project) through the fullbenchmark-stop -> resolve-subagent-traces.mjs -> build-run-metrics.mjschain in an isolated scratch dir — premature firings correctly skipped, genuine completion correctly proceeds.resolve-subagent-traces.mjsrun against a real orchestrator transcript — output matched that run's actual Performance Metrics table exactly.ccusage-wait.shrun against a real, already-indexed session id — immediate scoped match; against a synthetic non-existent id — correctly exhausts the 60s budget and falls back without hanging.turns/subagent_dispatches/tokens_by_model/cache_read_share_pct/models_used/tokens_by_agentall populate correctly, real per-TC tokens/tool_uses match the source transcript.reports/etc.) was touched during verification.🤖 Generated with Claude Code